# Lightweight Language Model

MiniLLM is a lightweight Chinese language model developed based on the bert4torch framework, covering the entire process from pre-training to instruction fine-tuning, with basic dialogue capabilities

An experimental text generation architecture that captures deeper contextual information through iterative latent processing

A 200M-parameter model based on the Qwen architecture, pretrained from scratch using the MiniPLM knowledge distillation framework

A text generation model employing mixed quantization techniques, with output and embedding tensors in f16 format and other tensors quantized using q5_k or q6_k. It has a smaller size than the standard q8_0 quantization format while maintaining performance comparable to the pure f16 version.

A 3B-parameter language model based on the Mamba architecture, supporting English text generation tasks.

This series of repositories open-sources reproductions of Meta AI's LLaMA and LLaMA 2 large language models, but with significantly reduced model sizes. The llama1_s experimental version contains 1.8 billion parameters, while the llama2_xs experimental version has only 460 million parameters.

A 124M-parameter language model based on the GPT-2 architecture, fine-tuned on 2.23B tokens of diverse data with improved text generation capabilities

A small Japanese language model based on GPT-NeoX architecture, supporting text generation tasks

This is a Japanese-pretrained ALBERT model that uses BertJapaneseTokenizer as its tokenizer, making Japanese text processing more convenient.

MiniLMv2 is a lightweight language representation model developed by Microsoft, achieving efficient performance through knowledge distillation technology.

A lightweight BERT model pre-trained on the BookCorpus dataset using knowledge distillation technology, with the hidden layer dimension reduced to 384 and 6 attention heads.

This is a sparse pre-trained model achieving 90% weight sparsity through one-shot pruning, suitable for fine-tuning on various language tasks.